Evaluating Digital Math Tools in the Field

Many school districts have adopted digital tools to supplement or replace teacher-led instruction, usually based on the premise that these tools can provide more personalized or individualized experiences for students and at lower cost. Rigorously evaluating whether such initiatives promote better student outcomes in the field is difficult as most schools and teachers are unwilling to enforce rigorous study designs such as randomized control trials. We used study designs that were feasible in practice to assess whether two digital math tools, eSpark and IXL, were associated with improvements in 3rd – 6th grade student test scores in math. We also investigated the resource requirements and costs of implementing eSpark and IXL to assess whether these tools represent a valuable use of resources. We find that while IXL is substantially less costly to implement than eSpark, its use is not significantly associated with students’ math performance

Evaluating Digital Math Tools in the Field

Many school districts have adopted digital tools to supplement or replace teacher-led instruction, usually based on the premise that these tools can provide more personalized or individualized experiences for students and at lower cost. Rigorously evaluating whether such initiatives promote better student outcomes in the field is difficult as most schools and teachers are unwilling to enforce rigorous study designs such as randomized control trials. We used study designs that were feasible in practice to assess whether two digital math tools, eSpark and IXL, were associated with improvements in 3rd – 6th grade student test scores in math. We also investigated the resource requirements and costs of implementing eSpark and IXL to assess whether these tools represent a valuable use of resources. We find that while IXL is substantially less costly to implement than eSpark, its use is not significantly associated with students’ math performance

Linguistic- and Acoustic-based Automatic Dementia Detection using Deep Learning Methods

Dementia can affect a person's speech and language abilities, even in the early stages. Dementia is incurable, but early detection can enable treatment that can slow down and maintain mental function. Therefore, early diagnosis of dementia is of great importance. However, current dementia detection procedures in clinical practice are expensive, invasive, and sometimes inaccurate. In comparison, computational tools based on the automatic analysis of spoken language have the potential to be applied as a cheap, easy-to-use, and objective clinical assistance tool for dementia detection. In recent years, several studies have shown promise in this area. However, most studies focus heavily on the machine learning aspects and, as a consequence, often lack sufficient incorporation of clinical knowledge. Many studies also concentrate on clinically less relevant tasks such as the distinction between HC and people with AD which is relatively easy and therefore less interesting both in terms of the machine learning and the clinical application. The studies in this thesis concentrate on automatically identifying signs of neurodegenerative dementia in the early stages and distinguishing them from other clinical, diagnostic categories related to memory problems: (FMD, MCI, and HC). A key focus, when designing the proposed systems has been to better consider (and incorporate) currently used clinical knowledge and also to bear in mind how these machine-learning based systems could be translated for use in real clinical settings. Firstly, a state-of-the-art end-to-end system is constructed for extracting linguistic information from automatically transcribed spontaneous speech. The system's architecture is based on hierarchical principles thereby mimicking those used in clinical practice where information at both word-, sentence- and paragraph-level is used when extracting information to be used for diagnosis. Secondly, hand-crafted features are designed that are based on clinical knowledge of the importance of pausing and rhythm. These are successfully joined with features extracted from the end-to-end system. Thirdly, different classification tasks are explored, each set up so as to represent the types of diagnostic decision-making that is relevant in clinical practice. Finally, experiments are conducted to explore how to better deal with the known problem of confounding and overlapping symptoms on speech and language from age and cognitive decline. A multi-task system is constructed that takes age into account while predicting cognitive decline. The studies use the publicly available DementiaBank dataset as well as the IVA dataset, which has been collected by our collaborators at the Royal Hallamshire Hospital, UK. In conclusion, this thesis proposes multiple methods of using speech and language information for dementia detection with state-of-the-art deep learning technologies, confirming the automatic system's potential for dementia detection

Essays on Applying Bayesian Data Analysis to Improve Evidence-based Decision-making in Education

This three-article dissertation aims to apply Bayesian data analysis to improve the methodologies that process effectiveness findings, cost information and subjective judgments with the purpose of providing clear, localized guidance for decision makers in educational resource allocation. The first article shows how to use a Bayesian hierarchical model to capture the uncertainty of the effectiveness-cost ratio. The uncertainty information produced by the model may inform the decision makers of the best- and worst-case scenarios of the program efficiency if it is replicated. The second article introduces Bayesian decision theory to address a subset of methodological barriers that hamper the influence of research on educational decision-making, including how to generalize or extrapolate effectiveness and cost information from the evaluation site(s) to a specific context, how to incorporate information from multiple sources, and how to aggregate multiple consequences of an intervention into one framework. The purpose of this article is to generate evidence of program comparison that applies to a specific school facing a decision problem by incorporating the decision-makers' subjective judgements and modeling their specific preference on multiple consequences. The third article proposes a randomized control trial to detect whether principals and practitioners update their beliefs on the effectiveness and cost of educational programs in the light of uncertainty information and localized evidence. Supplemented by a pilot qualitative study that guides decision makers to work on self-defined decision problems, the pilot testing of the experiment provides some evidence on the plausibility of using an experiment to identify the causal impact of research evidence on decision-making

Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos

Understanding dynamic hand motions and actions from egocentric RGB videos is a fundamental yet challenging task due to self-occlusion and ambiguity. To address occlusion and ambiguity, we develop a transformer-based framework to exploit temporal information for robust estimation. Noticing the different temporal granularity of and the semantic correlation between hand pose estimation and action recognition, we build a network hierarchy with two cascaded transformer encoders, where the first one exploits the short-term temporal cue for hand pose estimation, and the latter aggregates per-frame pose and object information over a longer time span to recognize the action. Our approach achieves competitive results on two first-person hand action benchmarks, namely FPHA and H2O. Extensive ablation studies verify our design choices. We will open-source code and data to facilitate future research

Education Leadership Data Analytics (ELDA): A White Paper Report on the 2018 ELDA Summit

Education Leadership Data Analytics (ELDA) is an emerging domain that is centered at the intersection of education leadership, the use of evidence-based improvement cycles in schools to promote instructional improvement, and education data science. ELDA practitioners work collaboratively with school and district leaders and teachers to analyze, pattern, and visualize previously unknown patterns and information from the vast sets of data collected by schooling organizations, and then integrate findings in easy to understand language and digital tools into collaborative and community-building evidence-based improvement cycles with stakeholders. In June of 2018, over 100 participants gathered for the Education Leadership Data Analytics Summit at Teachers College, Columbia University in New York City, including researchers, practitioners, policymakers, and funders. This report provides a summary of the central issues, themes, and recommendations for the future of the field that emerged from the discussions at the ELDA Summit event. These issues include building capacity in the field through incentivizing researcher practitioner partnerships, and providing conference and networking opportunities, professional development, certification, and ultimately degree programs to train ELDA researchers and practitioners. Additionally, a central focus of the ELDA field is equity, data security and privacy, in concert with open and FAIR data standards to develop and share de-identified data and tools across contexts. We conclude the report with a blueprint of possible skills and competencies needed for ELDA practitioner training and professional development and provide recommendations for next steps to help grow the field

Mobilizing Volunteer Tutors to Improve Student Literacy

This report evaluates the implementation, impacts, and costs of Reading Partners, a school-based early-intervention literacy program that relies on volunteer tutors. The work is a partnership between MDRC and CBCSE. The findings illustrate that the program is an efficient option for schools to consider in providing supplemental reading services to students who are not reading at grade-level

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research

We review the value of cost-effectiveness analysis for evaluation and decision making with respect to educational programs and discuss its application to early reading interventions. We describe the conditions for a rigorous cost-effectiveness analysis and illustrate the challenges of applying the method in practice, providing examples of programs for which we have estimated costs, but find effectiveness data lacking in comparability. We provide a demonstration of how cost-effectiveness analysis can be applied to two early reading programs: the Wilson Reading System and Corrective Reading. We use existing effectiveness data from an experimental evaluation in which the programs were implemented under similar conditions and the use of common outcomes measures for both programs yielded data that are comparable. We combine these data with cost data we collected using the ingredients method to calculate cost-effectiveness ratios for the alphabetics domain. A complete picture of the relative cost-effectiveness of each program could be provided if effectiveness metrics were available for fluency, vocabulary, and comprehension. We highlight the obstacles to applying cost-effectiveness analysis more frequently and recommend strategies for improving the availability of the requisite data